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System, Method, and Computer Program Product for Remote 
Graphics Processing 

Inventor: Phillip C. Keslin 

Background of the Invention 

Field of the Invention 

The present invention relates to computer graphics, and more particularly 
to graphics computing architecture. 

RelatedArt 

Today a user of a graphics processing host, such as an INFINITE 
REALITY system by Silicon Graphics, Inc. of Mountain View, California, runs 
a graphics application locally. Rendering in response to the graphics instructions 
is also performed locally. The distance from the host to a user can be extended 
to a distance of, for example, one to three kilometers by the use of fiberoptic 
cables to connect the user's monitor and keyboard with the graphics processing 
host. More important, the host is the dedicated resource of a single person. It can 
only be shared if someone else takes the place of the current user. While a 
dedicated resource is convenient for the user, it may not be economical. The cost 
of a graphics processing host is considerable; less than full usage of such a host 
lowers its cost-effectiveness. This lower cost-effectiveness could deter an 
organization from procuring a graphics processing host. 

Hence there is a need for a system and method by which additional users 
can utilize a remote graphics processing host. This would improve the cost- 
effectiveness of the host. Specifically, a system and method are needed where a 
user can execute a graphics application at a remote graphics processing host, 
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causing rendering to be performed at the host such that the resulting images are 
returned to the user. 

Summary of the Invention 

The invention described herein is a system, method, and computer program 
5 product for remote rendering of computer graphics. The invention includes a 

graphics application program resident at a remote server. The graphics application 
is invoked by a user or process located at a client. At the server, the invoked 
graphics application proceeds to issue graphics instructions. The graphics 
instructions are received by a remote rendering control system. Given that the 

10 client and server differ with respect to graphics context and image processing 

capability, the remote rendering control system modifies the graphics instructions 
in order to accommodate these differences. The modified graphics instructions are 
sent to graphics rendering resources, which produce one or more rendered images. 
Data representing the rendered images is written to one or more frame buffers. 

15 The remote rendering control system then reads this image data from the frame 

buffers. The image data is transmitted to the client for display or processing. In 
an embodiment of the invention, the image data is compressed before being 
transmitted to the client. In such an embodiment, the steps of rendering, 
compression, and transmission can be performed asynchronously in a pipelined 

20 manner. 

Features and Advantages 

The invention described herein has the feature of allowing a user to invoke 
a graphics application at a remote server. The invention also has the feature of 
allowing the graphics application to send graphics instructions to graphics 
25 rendering resources located with the server, so that rendering is performed at the 
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host and not at the user' s machine. The invention also has the feature of returning 
rendered image data to the user. 

The invention has the advantage of allowing a user to utilize graphics 
resources that are not co-located with the user, but are instead located at a 
5 distance from the user. The invention also has the advantage of allowing greater 

utilization of graphics resources, since users other than local users can take 
advantage of graphics resources. 

The foregoing and other features and advantages of the invention will be 
apparent from the following, more particular description of a preferred 
10 embodiment of the invention, as illustrated in the accompanying drawings. 



Brief Description of the Figures 



FIG. 1 illustrates the overall architecture of an embodiment of the 
invention. 

FIG. 2 is a block diagram of the remote rendering control system, 
15 according to an embodiment of the invention. 

FIG. 3 is a flowchart illustrating the method of the invention, according to 
an embodiment thereof. 

FIG. 4 is a flowchart illustrating the step of initialization, according to an 
embodiment of the invention. 
20 FIG. 5 is a flowchart illustrating the client / server handshake, according 

to an embodiment of the invention. 

FIG. 6 is a flowchart illustrating the graphics application's receipt of a 
client window, according to an embodiment of the invention. 

FIG. 7 is a flowchart illustrating the graphics application's receipt of a 
25 graphics context, according to an embodiment of the invention. 

FIG. 8 is a flowchart illustrating the binding of a server context to a server 
window, according to an embodiment of the invention. 
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FIG. 9 is a flowchart illustrating the step of imposing client parameters on 
the rendering process, according to an embodiment of the invention. 

FIG. 10 illustrates the pipelining of the rendering, compression, and 
transmission operations, according to an embodiment of the invention. 
5 FIG. 1 1 is a flowchart illustrating the remote rendering process from the 

perspective of the client, according to an embodiment of the invention. 

FIG. 12 illustrates the computing environment of the invention, according 
to an embodiment thereof. 



Detailed Description of the Preferred Embodiments 

10 A preferred embodiment of the present invention is now described with 

reference to the figures where like reference numbers indicate identical or 
functionally similar elements. Also in the figures, the left most digit of each 
reference number corresponds to the figure in which the reference number is first 
used. While specific configurations and arrangements are discussed, it should be 

1 5 understood that this is done for illustrative purposes only. A person skilled in the 

relevant art will recognize that other configurations and arrangements can be used 
without departing from the spirit and scope of the invention. It will be apparent 
to a person skilled in the relevant art that this invention can also be employed in 
a variety of other systems and applications. 



20 Contents 



I. Introduction 

II. Apparatus 

III. Method 

A. Server processing 
25 1. Initialization 

a. Client/server handshake 
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b. Client window 

c. Graphics context 

d. Context and server window 
2. Client parameters 

5 3. Rendering and transmission 

B. Client processing 

IV. Environment 

V. Conclusion 

/. Introduction 

10 The invention described herein is a system, method, and computer program 

product for remote rendering of computer graphics. The invention allows a user 
or process at a client computer to access remotely located rendering resources, 
such as a graphics processing host embodied in or co-located with a remotely 
located server. According to the invention, a graphics application executing at the 

15 server sends graphics instructions to a remote rendering control system. The 

remote rendering control system can be embodied in software running on the 
server. The instructions sent from the application are not sent to the client. Hence 
the client does no rendering. The instructions are modified by the remote 
rendering control system and sent to graphics rendering resources. Rendering 

20 consequently takes place at the graphics resources. The image data that is 

produced by the graphics resources are then sent to the client. 

II. Apparatus 

The basic architecture of an embodiment of the present invention is 
25 illustrated in FIG. 1. A user or process at a client machine 103 wishes to have 

rendering performed remotely. Accordingly, client 103 issues commands 107 to 
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a remotely located server 109. In an embodiment of the invention, connectivity 
between client 103 and server 109 is established and maintained through a 
communications network 115, such as the internet. Commands 1 07 are received 
at server 109 by a graphics application 120. In response to commands 107, 
5 application 120 generates graphics instructions 125. Graphics instructions may, 

for example, be in the OPENGL language. 

Graphics instructions 125 are sent to a remote rendering control system 
130. Remote rendering control system 130 accepts graphics instructions 125 and 
modifies them to create modified graphics instructions 135. An example of a 

10 remote rendering control system is the VIZSERVER software system produced 

by Silicon Graphics, Inc. As will described in greater detail below, the 
modifications to graphics instructions 125 are performed to take into account the 
differences between server 109 and client 103 with respect to their imaging 
processing capabilities and graphics contexts. In response to modified graphics 

15 instructions 135, graphics resources 140 render one or more images and return 

image data 1 45 to remote rendering control system 130. In an embodiment of the 
invention, image data 1 45 is then compressed to form compressed image data 1 50. 
Compressed image data 150 is then sent to client 103, via network 115. In 
alternative embodiments of the invention, compression is not implemented. 

20 A more detailed logical illustration of remote rendering control system 130 

is shown in FIG. 2. Remote rendering control system 130 accepts graphics 
instructions 125 through an interface 205 which is transparent to the graphics 
application 120. Graphics application 120 is effectively sending graphics 
instructions to local rendering resources instead of to client 103. Because of the 

25 transparency of interface 205, graphics application 1 20 need not be aware of this 

redirection. Transparent interface 205 performs a variety of modifications to 
graphics instructions 125, to produce modified graphics instructions 135. The 
modifications performed by transparent interface 205 will be described in greater 
detail below. Modified graphics instructions 135 are then sent to graphics 

30 resources 140 (not shown in FIG. 2) to effect rendering. 
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After rendering has been performed, remote rendering control system 130 
accepts the resulting image data 145. As described above, in an embodiment of 
the invention, image data 145 undergoes compression before transmission to client 
103. Compression of the data is performed by a data compression module 210, 
5 which produces compressed image data 150. Various methods of data 

compression are known to persons of ordinary skill in the art. In an embodiment 
of the invention, color cell compression is used. In alternative embodiments of the 
invention, data compression module 210 can perform alternative compression 
algorithms, such as the process described in U.S. Patent Application 09/458,01 1, 
1 0 "Image Data Compression and Decompression," incorporated herein by reference 

in its entirety. In an embodiment of the invention, data compression module 210 
is capable of performing any of a plurality of data compression algorithms. 

///. Method 

The overall process of an embodiment of the invention is illustrated in FIG. 

15 3. Process 300 starts at step 305. At step 310, a graphics session between a client 

and a remote server is initialized. Initialization step 310 includes the client's initial 
access to and protocol "handshake" with the server. In addition, step 310 includes 
the starting of the graphics application. Step 310 also includes reconciling the 
differences between the client and server with respect to graphics processing 

20 capability and graphics context. Such reconciliation is necessary to allow 

processing of an image at the server and display (or further processing) of the 
image at the client. Initialization will be described in greater detail below. 

In step 320, the graphics application generates graphics instructions for the 
remote rendering control system. In step 325, parameters specific to the client are 

25 imposed. This step includes, for example, interception by the transparent interface 

of function calls and references that occur in the graphics instructions, and the 
modification of those function calls and references to make them suitable to the 
client. In particular, adjustments may have to be made given that the client and 
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server can have different graphics processing capabilities and contexts. Step 325 
is described in greater detail below. 

In step 330, the remote rendering control system sends modified graphics 
instructions to the graphics resources. The modified instructions of this step 
correspond to the graphics instructions generated in step 320 and modified in step 
325. In step 335, the graphics resources render one or more images according to 
the modified instructions of step 330. 

In step 340, after rendering is completed, the remote rendering control 
system reads the resulting image data from a frame buffer. In an embodiment of 
the invention, the graphics application can have one or more specific instructions 
that force the reading of the frame buffer. If, for example, OPENGL is being 
used, the instruction "glflush" causes the frame buffer to be "flushed," i.e., read. 
The OPENGL instruction "glswapbuffers" also serves this purpose where two 
frame buffers are used. Here, the buffers will be swapped. A first buffer, to which 
image data has most recently been written, will be read. Writing, meanwhile, now 
takes place into the second buffer. Likewise, at the next "glswapbuffers" 
instruction, the second buffer will be read and the first buffer will begin receiving 
new image data. 

In step 345, the image data is enqueued for purposes of compression. In 
step 350, the image data is compressed so that in step 355 the image data can be 
transmitted to the client efficiently. In step 360, a determination is made as to 
whether additional rendering is necessary. If so, the process returns to step 320. 
If no additional rendering is required in step 360, the process concludes at step 
365. 

A. Server processing 
1. Initialization 
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The initialization step 310, according to an embodiment of the invention, 
is illustrated in greater detail in FIG. 4. The process begins at step 405. In step 
410, a user or process at the client machine, begins logging on to the server. As 
a result, in step 415a client/ server handshake takes place. The handshake includes 
5 the initial contact and protocol establishment between the server and client. Step 

415 also includes an agreement process between the client and server as to which 
data compression algorithm will be used, if there are more than one available. The 
client/server handshake process is described in greater detail below with respect 
to FIG. 5. In step 420, the client exports to the server a list of the client's 

10 graphics capabilities. A list of specific graphics capabilities is known as a visual. 

Examples of capabilities that may be included in a visual are the ability to perform 
stenciling or represent opacity. Another component of a visual is the color 
mapping range of a device, e.g., 48-bit versus 24-bit color representation. 

In step 425, the user or process starts a console window at the client 

15 computer. In step 430, the user or process starts the graphics application resident 

at the server. In step 435, the transparent interface to the graphics application 
opens the client display and the server display. In step 440, the transparent 
interface merges the visual of the client with the visual of the server. Because a 
graphics application typically deals with a single output device, the graphics 

20 application must see a single visual . This merger allows the graphics application 

to see a single visual, while in reality presenting visuals for both client and server. 
In step 445, the transparent interface associates the client display with the graphics 
application. In step 450, the transparent interface overlays the server visual with 
a routine that allows conversion of visual capabilities appearing in function calls 

25 of the graphics application. The routine converts these visual capabilities to 

capabilities appropriate to the client, i.e., capabilities in the client's original visual. 
The development and application of such a routine would be obvious to one of 
ordinary skill in the art given this description. 

In step 45 5, the client window is returned to the graphics application. This 

30 step includes the conversion, by the transparent interface, of the merged visual list 
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into a visual appropriate to the client. Step 455 also includes creation of an 
internal data structure for tracking the position of the client window in the client 
display. The process of receiving the client window is described in greater detail 
below with respect to FIG. 6. 
5 In step 460, the application receives a graphics context. This step, detailed 

below with respect to FIG. 7, includes definition of a server context by the 
transparent interface. Step 460 also includes the return of an internal context to 
the application. In step 465, the application binds the internal context to the client 
window, which has the effect of binding the server context to the server window. 
10 The initialization process concludes with step 470. 

a. Client / server h andshake 

An embodiment of the client/server handshake, step 415, is illustrated in 
greater detail in FIG. 5. This process begins with step 505. In step 510, the 
server connects to the client. In step 5 1 5 , the server (in particular, the application) 
15 identifies an address or source from which updates of image data are provided. 

It is from this source that updates will be sent to the client. In step 520, the server 
and client identify their respective available compression algorithms to each other. 
In step 525, the client chooses a compression algorithm and identifies it to the 
server. The process concludes with step 530. 

20 b. Client window 

The step by which a client window is returned to the application, step 455, 
is illustrated in greater detail in FIG. 6, according to an embodiment of the 
invention. The process begins with step 605. In step 610, the transparent 
interface produces a visual appropriate for the client from the merged visual list. 
25 This is done by taking only those visual capabilities which were originally the 

client's. Definition of such a visual is required for an image to be displayed 



SGI Confidential 



SKGF ref. 1452.3020000 
SGIref. 15-4-1034 



-11- 



properly at the client. In step 6 1 5, the transparent interface defines a window for 
the client. In step 620, the transparent interface creates an internal data structure 
for tracking the position of the client window on the client display. In step 625, 
the transparent interface returns the client window to the application. The process 
5 concludes with step 630. 

c. Graphics context 

An embodiment of the process by which a graphics context is returned to 
the application, step 460, is illustrated in greater detail in FIG. 7. The process 
begins with step 705. In step 710, the transparent interface produces a server 

1 0 visual from the merged visual. This is done by taking only those visual capabilities 

which were originally the server's. In step 715, the transparent interface defines 
a server context. The server context represents the context settings to be applied 
in rendering for the server's display. In step 720, the transparent interface returns 
an internal context to the application program, where the internal context includes 

1 5 the server context and any additional information that may be necessary to allow 

the transparent interface to properly interpret references to the server context. An 
example of such additional information is the identity of specific visual capabilities 
that correspond to a given element of the server context. The process concludes 
with step 725. 

20 d. Context and server window 

The step of binding a context to a server window, step 465, is illustrated 
in greater detail in FIG. 8, according to an embodiment of the invention. The 
process begins with step 805. In step 810, the transparent interface extracts the 
server context from the internal context. In step 815, the transparent interface 
25 requests a window allocation from the session manager. The session manager is 

logic (e.g., software) responsible for managing the allocation of resources in a 
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graphics session. The implementation of a session manager is known to persons 
of ordinary skill in the art. In step 820, the transparent interface binds the server 
context to the server window. The process concludes with step 825. 



2. Client parameters 



5 Returning to the overall process 300 illustrated in FIG. 3, once 

initialization is completed in step 3 1 0, the graphics application generates graphics 
instructions in step 320, as discussed above. This is followed by step 325, the step 
of imposing client parameters on the rendering process. An embodiment of step 
325 is illustrated in greater detail in FIG. 9. The process begins with step 905. 

10 In step 9 1 0, the transparent interface intercepts graphics instruction function calls 

that include a visual. In step 9 1 5, the transparent interface converts the visuals to 
corresponding client visuals. In step 920, the transparent interface intercepts 
every graphics instruction reference to a context. In step 925 the transparent 
interface converts the reference to a reference consistent with the client' s context. 

15 The process concludes at step 930. 



3. Rendering and transmission 



Steps 340 through 355 (rendering, enqueuing, compression, and 
transmission) are collectively illustrated in FIG. 10, according to an embodiment 
of the invention. This figure illustrates how blocks of image data are rendered, 

20 compressed, then transmitted. In the illustrated embodiment, these operations are 

performed in pipeline fashion. The rendering, compression, and transmission steps 
are identified collectively as processes 1005. These steps take place sequentially 
during successive blocks of time 1010. During time to, a block A of image data 
is rendered. During the next interval, time t u block A is compressed. 

25 Simultaneously, the next segment of image data, block B, is rendered. At the next 

time interval, block A is transmitted after having been compressed during the 
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previous time period. While block A is being transmitted, block B is compressed. 
Simultaneously, the next block of image data, block C, is being rendered. 
Successive blocks of image data are processed in this manner. 

In an embodiment of the invention, the steps of rendering, compression, 
and transmission are asynchronous. In such an embodiment, the compression of 
block B may not be completed at the same time as the rendering of block C, 
referring to the example above. To deal with this, blocks of image data may be 
discarded. Any given step will take only the most recent block from the previous 
step. If, for example, rendering of block B takes place faster than the compression 
of block A, the compression module is not yet ready for block B when rendering 
of block B is done. Block B will therefore be discarded. The compression module 
will finish compressing block A, then wait for the next (and newest) block to 
emerge from rendering. Multiple blocks may be discarded if rendering is 
significantly faster than compression. The transmission process will likewise 
accept only the newest compressed block of image data. If compression of a 
block is completed before the previous block has been transmitted, that 
compressed block will be discarded. 

B. Client processing 

An embodiment of the remote rendering process from the perspective of 
the client is illustrated in FIG. 11. The process starts with step 1105. In step 
1110, the client/server handshake takes place, as described above. In step 1115, 
the client sends one or more commands to the graphics application to begin 
execution. In step 1 1 20, after the graphics resources at the server have completed 
rendering, the client receives image data via the network. In an embodiment of 
the invention, the image data is compressed for purposes of transmission 
efficiency. In step 1 1 25, the image data is decompressed at the client and in step 
1 130, the appropriate image is drawn to a window at the client. In step 1 135 a 
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determination is made as to whether an additional image is to be drawn. If so, the 
process continues at step 1 120. Otherwise the process concludes at step 1 140. 

IV. Environment 

Server 109 of the present invention may be implemented as a computer 
5 system or other processing system. Remote rendering control system 130 and 

graphics application program 120 can be implemented as computer programs 
executing on such a computer system. Graphics resources 140 can be any type of 
graphics subsystem linked to a communication infrastructure 1206, such as a bus 
or network. An example of such a computer system 1200 is shown in FIG. 12. 

10 The computer system 1200 includes one or more processors, such as processor 

1204. The processor 1204 is connected to communication infrastructure 1206. 
Various software implementations are described in terms of this exemplary 
computer system. After reading this description, it will become apparent to a 
person skilled in the relevant art how to implement the invention using other 

15 computer systems and/or computer architectures. 

Computer system 1200 also includes a main memory 1208, preferably 
random access memory (RAM), and may also include a secondary memory 1210. 
The secondary memory 1210 may include, for example, a hard disk drive 1212 
and/or a removable storage drive 1214, representing a floppy disk drive, a 

20 magnetic tape drive, an optical disk drive, etc. The removable storage drive 1214 

reads from and/or writes to a removable storage unit 1218 in a well known 
manner. Removable storage unit 1218, represents a floppy disk, magnetic tape, 
optical disk, or other storage medium which is read by and written to by 
removable storage drive 1214. As will be appreciated, the removable storage unit 

25 1218 includes a computer usable storage medium having stored therein computer 

software and/or data. 

In alternative implementations, secondary memory 1210 may include other 
means for allowing computer programs or other instructions to be loaded into 
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computer system 1200. Such means may include, for example, a removable 
storage unit 1222 and an interface 1220. Examples of such means may include a 
program cartridge and cartridge interface (such as that found in video game 
devices), a removable memory chip (such as an EPROM, or PROM) and 
5 associated socket, and other removable storage units 1222 and interfaces 1220 

which allow software and data to be transferred from the removable storage unit 
1222 to computer system 1200. 

Computer system 1200 may also include a communications interface 1 224 . 
Communications interface 1224 allows software and data to be transferred 

10 between computer system 1200 and external devices. Examples of 

communications interface 1 224 may include a modem, a network interface (such 
as an Ethernet card), a communications port, a PCMCIA slot and card, etc. 
Software and data transferred via communications interface 1 224 are in the form 
of signals 1228 which may be electronic, electromagnetic, optical or other signals 

15 capable of being received by communications interface 1 224. Commands 1 07 and 

compressed image data 1 50 are examples of signals 1228. These signals 1 228 are 
provided to communications interface 1224 via a communications path (i.e., 
channel) 1226. This channel 1226 carries signals 1228 and may be implemented 
using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link and 

20 other communications channels. 

In this document, the terms "computer program medium" and "computer 
usable medium" are used to generally refer to media such as removable storage 
units 1218 and 1222, a hard disk installed in hard disk drive 1212, and signals 
1228. These computer program products are means for providing software to 

25 computer system 1200. 

Computer programs (also called computer control logic) are stored in main 
memory 1208 and/or secondary memory 121 0. Computer programs may also be 
received via communications interface 1224. Remote rendering control system 
130 and graphics application program 120 can be implemented as computer 

30 programs. Such computer programs, when executed, enable the computer system 
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1200 to implement the present invention as discussed herein. In particular, the 
computer programs, when executed, enable the processor 1204 to implement the 
present invention. Accordingly, such computer programs represent controllers of 
the computer system 1200. Where the invention is implemented using software, 
5 the software may be stored in a computer program product and loaded into 

computer system 1200 using removable storage drive 1214, hard drive 1212 or 
communications interface 1224. 



V. Conclusion 



While various embodiments of the present invention have been described 
1 0 above, it should be understood that they have been presented by way of example, 

and not limitation. It will be apparent to persons skilled in the relevant art that 
various changes in detail can be made therein without departing from the spirit and 
scope of the invention. Thus the present invention should not be limited by any 
of the above-described exemplary embodiments, but should be defined only in 
15 accordance with the following claims and their equivalents. 



SGI Confidential 



SKGF ref. 1452.3020000 
SGIref. 15-4-1034 



-17- 



What Is Claimed Is: 



1 1. A system for providing a client with access to remote graphics 

2 rendering resources, comprising: 

3 a remote rendering control system that receives graphics 

4 instructions, generates modified graphics instructions on the basis of said graphics 

5 instructions, and outputs said modified graphics instructions to said graphics 

6 rendering resources. 

1 2. The system of claim 1 , wherein said remote rendering control 

2 system comprises a transparent interface to said graphics application, and wherein 

3 said transparent interface supports initialization of a graphics rendering session and 

4 accommodates client parameters during said graphics rendering session. 

1 3. The system of claim 1, wherein said remote rendering control 

2 system comprises a data compression module that compresses said image data 

3 prior to sending said image data to said client. 

1 4. The system of claim 1, wherein said remote rendering control 

2 system receives image data generated by said graphics rendering resources on the 

3 basis of said modified graphics instructions, and sends said image data to said 

4 client. 

1 5. The system of claim 1, wherein said remote rendering control 

2 system receives graphics instructions from a graphics application program. 

1 6. A method of remote graphics rendering on behalf of a client, 

2 comprising the steps of: 

3 (A) initializing a graphics rendering session; 
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4 (B) starting a graphics application on the basis of a command 

5 from the client; 

6 (C) generating graphics instructions; 

7 (D) imposing client parameters to produce modified graphics 

8 instructions; 

9 (E) sending the modified graphics instructions to graphics 

10 rendering resources; 

1 1 (F) rendering graphics on the basis of the modified graphics 

12 instructions to produce image data in one or more frame buffers; 

13 (G) reading image data from the one or more frame buffers; 

14 (H) enqueuing the image data; and 

15 (I) transmitting the image data to the client. 

1 7. The method of claim 6, further comprising the step of: 

2 (J) compressing the image data, 

3 performed after step (H) and before step (I). 

1 8. The method of claim 7, wherein steps (F), (J), and (I) are 

2 performed in pipeline fashion. 

1 9. The method of claim 8, wherein steps (F), (J), and (I) are 

2 asynchronous. 

1 10. The method of claim 6, wherein step (A) comprises the steps of: 

2 (i) performing a client / server handshake; 

3 (ii) receiving a client visual from the client; 

4 (iii) after a user at the client opens a console window at the 

5 client and starts the graphics application, opening client and server displays; 

6 (iv) merging the client visual with a server visual to form a 

7 merged visual list; 
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8 (v) associating the client display with the graphics application; 

9 (vi) overlaying the server visual list with a transparent interface 

10 routine; 

11 (vii) enabling the return of a client window to the graphics 

12 application; 

13 (viii) enabling the return of an internal context to the graphics 

14 application; and 

15 (ix) binding a server context to the server window. 

1 11. The method of claim 1 0, wherein step (vii) comprises the steps of: 

2 (a) converting the merged visual list into a visual 

3 appropriate for the client; 

4 (b) defining the client window; 

5 (c) creating an internal data structure for tracking the 

6 displayed location of the client window; and 

7 (d) returning the client window to the graphics 

8 application. 

1 12. The method of claim 1 0, wherein step (viii) comprises the steps of: 

2 (e) converting the merged visual list into a visual 

3 appropriate for the server; 

4 (f) creating a server context; and 

5 (g) returning an internal context to the application. 

1 13. The method of claim 1 0, wherein step (ix) comprises the steps of: 

2 (h) extracting a server context from the internal 

3 context; 

4 (i) requesting a window allocation from a session 

5 manager; and 

6 (j) associating the server context with a server window. 
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1 14. The method of claim 6, wherein step (D) comprises the steps of: 

2 (x) intercepting every function call that includes a visual 

3 capability; 

4 (xi) converting the visual capability to a corresponding client 

5 visual capability; 

6 (xii) intercepting every reference to a graphics context; and 

7 (xiii) converting every reference to a graphics context to a 

8 reference to a graphics context of the client. 

1 15. A computer program product comprising a computer usable 

2 medium having computer readable program code that enables remote graphics 

3 rendering on behalf of a client, said computer readable program code comprising: 

4 first computer readable program code logic for causing a server to 

5 initialize a graphics rendering session; 

6 second computer readable program code logic for causing the 

7 server to start a graphics application on the basis of a command from the client; 

8 third computer readable program code logic for causing the server 

9 to generate graphics instructions; 

1° fourth computer readable program code logic for causing the 

1 1 server to impose client parameters to produce modified graphics instructions; 

1 2 fifth computer readable program code logic for causing the server 

13 to send the modified graphics instructions to graphics rendering resources; 

14 sixth computer readable program code logic for causing the 

15 graphics rendering resources to render graphics on the basis of the modified 

16 graphics instructions to produce image data in one or more frame buffers; 

17 seventh computer readable program code logic for causing the 

18 server to read image data from the one or more frame buffers; 

19 eighth computer readable program code logic for causing the 

20 server to enqueue the image data; and 
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21 ninth computer readable program code logic for causing the server 

22 to transmit the image data to the client. 

1 16. The computer program product of claim 15, said computer 

2 readable program code further comprising: 

3 tenth computer readable program code logic for causing the server 

4 to compress the image data. 

1 17. The computer program product of claim 15, wherein said first 

2 computer readable program code logic comprises: 

3 (i) computer readable program code logic for causing the server to 

4 participate in a client / server handshake; 

5 (ii) computer readable program code logic for causing the server 

6 to receive a client visual from the client; 

7 (iii) computer readable program code logic for causing the server 

8 to open client and server displays after a user at the client opens a console window 

9 at the client and starts the graphics application; 

10 (iv) computer readable program code logic for causing the server 

11 to merge the client visual with a server visual to form a merged visual list; 

12 (v) computer readable program code logic for causing the server 

13 to associate the client display with the graphics application; 

14 (vi) computer readable program code logic for causing the server 

15 to overlay the server visual list with a transparent interface routine; 

1 6 (vii) computer readable program code logic for causing the server 

17 to enable the return of a client window to the graphics application; 

1 8 (viii) computer readable program code logic for causing the server 

19 to enable the return of an internal context to the graphics application; and 

20 (ix) computer readable program code logic for causing the server 

21 to bind a server context to the server window. 
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1 18. The computer program product of claim 17, wherein said computer 

2 readable program code logic (vii) comprises: 

3 (a) computer readable program code logic for causing the 

4 server to convert the merged visual list into a visual appropriate for the client; 

5 (b) computer readable program code logic for causing the server 

6 to define the client window; 

7 (c) computer readable program code logic for causing the server 

8 to create an internal data structure for tracking the displayed location of the client 

9 window; and 

10 (d) computer readable program code logic for causing the server 

11 to return the client window to the graphics application. 

1 19. The computer program product of claim 1 7, wherein said computer 

2 readable program code logic (viii) comprises: 

3 (a) computer readable program code logic for causing the 

4 server to convert the merged visual list into a visual appropriate for the server; 

5 (b) computer readable program code logic for causing the 

6 server to create a server context; and 

7 (c) computer readable program code logic for causing the 

8 server to return an internal context to the application. 

1 20. The computer program product of claim 1 7, wherein said computer 

2 readable program code logic (ix) comprises: 

3 (a) computer readable program code logic for causing the server 

4 to extract a server context from the internal context; 

5 (b) computer readable program code logic for causing the server 

6 to request a window allocation from a session manager; and 

7 (c) computer readable program code logic for causing the server 

8 to associate the server context with a server window. 
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1 21. The computer program product of claim 15, wherein said fourth 

2 computer readable program code logic comprises: 

3 (i) computer readable program code logic for causing the 

4 server to intercept every function call that includes a visual capability; 

5 (ii) computer readable program code logic for causing the 

6 server to convert the visual capability to a corresponding client visual capability; 

7 (iii) computer readable program code logic for causing the 

8 server to intercept every reference to a graphics context; and 

9 (iv) computer readable program code logic for causing the 

1 0 server to convert every reference to a graphics context to a reference to a graphics 

1 1 context of the client. 
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System, Method, and Computer Program Product for Remote 
Graphics Processing 

Abstract 

A system, method, and computer program product are provided for remote 
5 rendering of computer graphics. The system includes a graphics application 

program resident at a remote server. The graphics application is invoked by a user 
or process located at a client. The invoked graphics application proceeds to issue 
graphics instructions. The graphics instructions are received by a remote 
rendering control system. Given that the client and server differ with respect to 

10 graphics context and image processing capability, the remote rendering control 

system modifies the graphics instructions in order to accommodate these 
differences. The modified graphics instructions are sent to graphics rendering 
resources, which produce one or more rendered images. Data representing the 
rendered images is written to one or more frame buffers. The remote rendering 

1 5 control system then reads this image data from the frame buffers. The image data 

is transmitted to the client for display or processing. In an embodiment of the 
system, the image data is compressed before being transmitted to the client. In 
such an embodiment, the steps of rendering, compression, and transmission can 
be performed asynchronously in a pipelined manner. 

20 P:\USERS\EYEE\Sgil452\14S2.302\A276-S7.wpd 



SGI Confidential 



SKGF ref. 1452.3020000 
SGIref. 15-4-1034 



100 

J 




FIG. 1 



0002-28.vsd1 



200 

J 



Graphics 
instructions 
125 



Transparent interface 205 


(TIF) 










Data 




compression 




module 




210 



Modified 
graphics 
instructions 
135 



To 
client 
103 



Compressed 
Image data^ 
150 



Image 
data 
145 



FIG. 2 



0002-28.vsd2 




Initialization of session 



Application generates graphics 
instructions for remote rendering 
control system 



Impose client parameters 



Remote rendering control system 
sends modified instructions to 
graphics resources 



Rendering 



Remote rendering control system 
reads image data from frame buffer 



Enqueuing 



-345 



Image data compressed 



^350 



300 



Compressed image data 
transmitted to client 



-355 



360 



Yes 




No 



FIG. 3 



0002-28.vsd3 



Start 



-405 



User at client begins logging on 
to server 



-410 



310 



J 



Client/server handshake -—415 



Client exports visual to server 



User starts console window 



—425 



User starts application 



TIF opens client display and 
server display 



-435 



TIF merges visuals of client and 
server 



-440 



TIF associates client display 
with application 



-445 



TIF overlays the merged visuals 
with TIF routine 



Application receives client 
window 



-455 



Application receives internal 
context 



Application binds server context 
to server window 



Return 



FIG. 4 



0002-28.vsd4 



415 



Start W505 







Server connects to client 







Server identifies source of 
updates 






Server and client identify 
available compressors 






Client identifies chosen 
compressor 







-515 



-520 



-525 



J 



End 



FIG. 5 



0002-28.vsd5 



455 

J 

Start ) — 605 







TIF produces visual appropriate 
for client from merged visual list 






TIF defines client window 






TIF creates internal structure for 
tracking client window 






TIF returns client window to 
application 







-615 



Return ) — 630 



FIG. 6 



TIF produces server visual from 
merged visual list 







TIF defines server context 






TIF returns internal context to 
application 



Return V-725 



FIG. 7 



465 



Start 







Application extracts server 
context from internal context 






Application requests window 
allocation from the session 
manager 






Application binds server context 
to server window 







-820 



-825 



FIG. 8 



j 



0002-28.vsd8 



325 



Start W905 







TIF intercepts every function 
call that has a visual capability 
associated with it 






TIF converts to client visual 
capability 






TIF intercepts every reference to 
graphics context 






TIF converts to client's context 







-910 



Return ) — 930 



J 



FIG. 9 



0002-28.vsd9 



1000 

J 



Processes 1005 ► 

Render Compress Transmit 



Time 


to 


A 






1010 














ti 


B 


A 








t 2 


C 


B 


A 



FIG- 10 



0002-28. vsd 11 



1100 

J 



Start ) — 1105 



Client/server handshake 


— 1 110 








Send command(s) to application 


— — -1 115 








Receive compressed image data 
from network 


— 1120 








Decompress image data 


—1125 








Draw image to window 


—1130 



Yes 




FIG. 11 

0002-28. vsd10 



Processor 1204 



Computer System 1200 



Main Memory 1209 



Communication 
Infrastructure 
1206 



Secondary Memory 1210 




Hard Disk Drive 1212 








Removable Storage Drive 1214 












Interface 1220 











Removable Storage 
Unit 1218 



Removable Storage 
Unit 1222 



A 
V 



Communications 
Interface 1224 




Communications Path 1226 



FIG. 12 



0002-28.vsd12 



Declaration for Patent Application 



Docket Number: 15-4-1034.00 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name. 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if 
plural names are listed below) of the subject matter that is claimed and for which a patent is sought on the invention entitled. 
System, Method, and Computer Program Product for Remote Graphics Processing, the specification of which is attached 
hereto unless the following box is checked: 

□ was filed on ; 

as United States Application Number or PCT International Application Number ; and 

was amended on (if applicable). 

I hereby state that I have reviewed and understand the contents of the above identified specification, including the claims, as 
amended by any amendment referred to above. 

I acknowledge the duty to disclose information that is material to patentability as defined in 37 C.F.R. § 1 .56. 

I hereby claim foreign priority benefits under 35 U.S.C. § 119(a)-(d) or § 365(b) of any foreign application(s) for patent or 
inventor's certificate, or § 365(a) of any PCT international application, which designated at least one country other than the 
United States listed below, and have also identified below any foreign application for patent or inventor's certificate, or PCT 
international application having a filing date before that of the application on which priority is claimed. 

Prior Foreign Application(s) Priority Claimed 

nYes nNo 

(Application No.) (Country) (Day/Month/Year Filed) 

□ Yes nNo 

(Application No.) (Country) (Day/Month/Year Filed) 

I hereby claim the benefit under 35 U.S.C. § 1 19(e) of any United States provisional application(s) listed below. 



(Application No.) (Filing Date) 



(Application No.) (Filing Date) 

I hereby claim the benefit under 35 U.S.C. § 120 of any United States application(s), or under § 365(c) of any PCT 
international application designating the United States, listed below and, insofar as the subject matter of each of the claims 
of this application is not disclosed in the prior United States or PCT international application in the manner provided by the 
first paragraph of 35 U.S.C. §112,1 acknowledge the duty to disclose information that is material to patentability as defined 
in 37 C.F.R. § 1.56 that became available between the filing date of the prior application and the national or PCT 
international filing date of this application. 



(Application No.) (Filing Date) (Status - patented, pending, abandoned) 



(Application No.) (Filing Date) (Status - patented, pending, abandoned) 



-Page 1 of 2- 



Appl. No. (to be assigned) 
Docket No. 15-4-1034.00 



Send Correspondence to: 



Sterne, Kessler, Goldstein & Fox p.l.l.c. 
1 100 New York Avenue, N.W. 

Suite 600 
Washington, D.C. 20005-3934 



Direct Telephone Calls to: 



(202) 371-2600 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may jeopardize the validity of the application or any patent issued 
thereon. 




Full name of sole or first 



San Jose, California 



Post Office Address 



7483 Phinney Way, San Jose, California 95 139 



Full name of second 



Signature of second inventor 



Post Office Address 



Full name of third 



Post Office Address 



-Page 2 of 2- 



